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^ , Prbgram evaluaticfe methodologies ar;e a family of investigatory ,> 
Strategies which have ^been d(aveloped ^in recfent years in ^response to.-the 
need for procedure^ for'*; generating reiiabLe information *as to the capa- 
bilities of h^ni^ih s^A^^Je^ Human seftvice programs in general,; 
and corrections prograihsi iri 'particular , are intended to generate ,a variety " 
of beneficial effects in the clientele for wh6m they are ■ designed . 

" >: ""^^ ■■■^ . ^ ' • ' . , ^ ■ , ' . " ' 

Evaluative data is needed to appraise the extent to which such programs 

' " , ^ • * • i ^ , • ■ ■ • 

are capable of 'j^rodufcing th6pe effects in reliable fashion, and to 

detail/what is required in order to produce them. The procedures are 

intended to be 'used in Iterative fashion, so that b};^ a succe^sipn of 

attempts, suc'h programs can be moved over time in the direction of hi- 

effect, lo-co9t service systems. 

Experience wi^th this family of procedures as they have evolved 
to date, however, hJs disclosed multiple dlJpEiculties attendant upon 
thestjr^ use, doubt as to the validity bf the information which tho^ generate, 
ano" a legitd-tnate questioning as to the impact which the findings, have 
upon the upgrading and improvement of human service programs. Indeed, 
it has become customary of late to enumerate the many problems with 
which program evaluations have cotne to be plagued, and these are often 
accompanied with warnings to the effect that the entire effort will 
becomfe discredited and the support of human service programs in the future 
endangered, if these do not become corrected. 

Ciagnoses as to the nature of the difficulties are highly varied, 
and corre^por^dlngly suggestion^ for remedy are equally varied as well. 
An ins|j|j^tjb;i^f the current literature in human service program evaluation 
^11 serv-d to reyeal that heterogeneity. 



'^However, a number of critics of our contemporary^ procediiries 
^identify the^ fault as residing withir^ thd processes of pA^^gram development 
and evaluation which have come to be followed In the^ 'human service , sectoral . 
It is their contention that the manner in which we have attempted to - 
.design, develop, test and operate innovative processes and programs i^ ^ 
not one destinedf- to be particularly productive or successful in. producing 
the kinds of client-programs which people need and .want, ,' They note that 
alternative strategies are available to those we currently employ. . Tor emost 
among .these is the innovative process for program development and implex- 
mentation which has been in estat4\Lshed use in industrial,, agricultural 
military, and medical* contexts. 'The diffierences between what that 
process can generate in the. way of results versus* those^ which we currently 
pursue in the human services area can be illustrated at the level of 
concrete service delivery, viz * the human service agency. 

We can begin with an hypothetic director of some human se;rvice 
program - which kind doesn't really matter. He or shetmighfc be the' 
director of a Mental Health Center, head of a Probation Department, or . " ' 
someone in charge of a C^ilidren^s Services A-gency. Let us suppose that ouf 
director is interested in^upgrading aind improving the operation of his 



program or services in a variety of ways.>^ He needs to introduce some 
innovations into his service system. ^ * ^ ' 

It occurs to him that something needs to be done about the secret'arial 
support system that serves his prof essional staff , since it turns out 
that it is inevitably lagging behind in report preparation, and incapable 
of keeping abreast of the work of the staff, despite a stable of hi-speed 
typists. He needs greater producti^ityy..^^means of an alternative set 



of procedures, but atv no^a^preciable increase; costs. Cooversion of 
a segment of his secretarial system to ^ mag-card unit seems a possible 
solution. \ ^ ♦ ^ 

Contafct with a technical representative of a specialty firm can 

• ■ . ■ ■ 

put him in a position to decide whether or TiQt the installation of such a 
unit could be expected to upgrade the delivery capabili^pf his agency; 
he can make such decisions because he }ls provided with an information base 
for that purpo^. The specialists with -whom he consults can inform him 
in advance of the space requirements for such a unit, the nature of its 
performance capability, or its durability, the personnel competencies whicji 
are needed, its costs with regard^o installation, maintenance, and operation, 
and a way of estimating cost-savit>|^\ ^Hevis^i^,^ position to make a go/no-go 
decision? If he makes the decision to* proceed technical * representation ' 
will then providt: equipment in „place, 'manuals for its operation and 

procedures for.it'S use, specifications for its management and security, 

' ^ 

a reeord system 'for data collection and analysis so thaj? *the, agency 
director can audit the performance 'of the unit along With the cost of its 




ration, a training package f oir technicians^* isupervised trial operations 

' ^ . \ ■ . ■ ' ' V 

trainees up to a floor specif ii^tion, a set of procedures in order to 
forestall and correct for the degradation^^f^he service system over time, 
availability of continuing technical assistance, monitoring', and repair, 
and a contractual*^ agreement specifying whajT^^will be prpvided . Our service 
manager may ' not have a detailed understanding of how all of this copld 
have come about; but he does know that whatever the process was, he and 
his Agency are equipped with all of the means needed to activate tbe ' . 
new /technology i'ntq his service agency and make it productive. He also 
knows that he and his secretaries could not have designed developed, 



and testfed such a process themselves, eVen with a federal grant Intended, 
,; to promote the introduction of innovative programs' and procedures into 

' ; V : * . ■ . * ' ' , . ; ■ ^ 

an. agency 'such as h'e has. Finally, he knows that having been spared such 

a task .of innovating;a new ^t ethnology , he and his staff can focus upon their 

. * ' ' - ^ \ • -° . ■ ■ : 

, principal milss'ioa, which is that of providing services to, their clientele i 

• Suppose he now turns to his' service programs themselves. He knows 

• % ..... . ■ . ' ■ ' 

that technical improvefnents need to be introduced here ^ as well.' There" 

, are alcoholic patients with whom current services do not succeed, or \ 

. adjudicated delinquents ^whose needs appear to bewilder his staff,* or 

« abusive parents for whom no services are currently being performed. How 

is our director to proceed in these instances? If he does not already 

know, he very quickly discovers that the situation here is very different. 

, He will ]^arn of multiple alternative programs . from other agency directors, - . 

from l^perusal of the program literature, or from conversations with 

consultants of one sort or^another* but it will be difficult for him to 

ohoose from among them because he will have no cle^r idea what th'^e 

technical capabilities of most ,of these programs actually are. The 

programs and procedures which he inspects and from-Vhich he may endeavor to 

choose are ^in a comparatively crude stage of development, and the way in 

.which their development has been attempted has not been such as to generate 

the kinds of information he needs to have in ordei;, to make an effective 

decision. He will have no clear fix on what kinds of p'ersonnel "are required 

to implement them, no adequate documentation a^ to just what the procedures 

can do in terms of their range of applicability, their success/failure 

ratios, or indie?ations with respect to deleterious side-effects, no clear 

• (idea of costs, no way tp monitar the implementation, no. ready mechanisms by 

which to effect year-end; cost and performance accounting, and on and on it goes 



With a few c^the programs he will discovet that some evaluative" 
testing will have been dbne. .£ut the information generated' 1,y such 
evaluative studies will upon^clo'se inspection prove to be extremely 
scant, stemming from phfe irtnumera^jle compromises which^ t^e evaluators were 
forced to make as to the variety of information they pould collect, and' 
the procedures they could "employ in order to Collect it. 

And yet circumstances call for a selection to be made", arid thus 
.our service director will be led to making some sort of choice on other ' 
than solid informational grounds: he may choose whichever prJgram- 
sequences appear to be consistent with his o^ ideological persuasion, 
or which are acceptable to his staff; he may respond to the rhetoric pf 
some program-advocates as o^^ed to others; he may .select in terms of 
curreftt popularity or prof essixDnal consensus; or he can set aside time, 
money, and personnel from the day-to-day task of providing for plients in 
need. In ckdev to cqnceive, design, develop, implement, and seek to 
evaluate a program "of their own, and applying 'for grant support for the ■ 
purp9se.- Although there appears to be a certain amount of prestige 
associated with being awarded such a grant since to marif it looks like an. 
endorsement of the worth of an agency and seems to add Resources to . 
that portion o^ the service sector, at the same time it will prove to be - 
a mixed blessing at best, and a seriblis disruption "of the program of". • 
services of his agency at worst; and the evaluation component .which is 
included, sufficient perhaps to produce a_grant award, will prove no ^ 
more adequate in generating, hard inforamtion of a comprehensive; and detailed 
sort. than those evaluative studies h6 had earlier reviewed. 



How is it that we have such a marked constrast l^tween what our 

service manager can do in the way of upgrading and improving procedur'es^dn 

one sphere as opposed to the other? A quick answer might be that he is 

dealing with very different sets. of things - devices and machines on the ' 

'one. hand, and people endeavoring to help other people on the other; or 

one might observe. that efforts to introduce technical sophistication.^nto 

human service programs is a relatively recerit development and we are still 

in the process of learning how best to do it. But, answers which say that 

» ' ' ■ ' 

the task is harder, or' that we have been at it a much shorter time, serve 
to divert one's att^ntiort away from a very important factor - and that \ 

that the process by which technical innovations become developed, 
tested and. subsequently utilized in one, arena is so very different from ' 
the process which we have become accustomed to following in the other. 
Following different processes, ohe can anticipate different outcomes. ' - 

The term innovation, as it is used iri'the fields of engineering, 
agriculture or in m:edical technology, -refers to the sequence' of%;t,eps 
by which knowledge and technique become transformed and combined into - 
an action-sequence capable of generating effects deemed to be of utility 
to people. It is a procedural sequence which ifcself has undergone 
development,. testing, and refinement' 'ov6r k lengthy period of time. It 
■has become widely used because of its capability for generating ou£tomes ' 
which people^'need and wa.nt,_aiid- to produce those effects with higher . 
probability of success, less waste of resources aiid . fewer chances of error^' 
than alternative strategies .which might be pursued' iftstead. 

There are a variety of w.ays in, which the cc^mplex innovation process 
can be represented. A short-hand' way of explicating the process is 
represented in the accompanying figure, which endeavors to lay out in 



v.. 



sequent?.al. order the functions which need to performed in order to 

pursue the process. Each step in the process can, ^of course, be broken/ 

down. iTfitol finer incremental steps, and a detailed representation of the ' 
• ■ . I ■ ■ . . • . r ■ 

process would pi^oceed to do so. Hdwever, the figure is intended to provide • 

the major phases which are entailed', whether one is concerned with the 
design and development:' of an innovative device, a product, a program of / 
ac^on, or an entire orgnaiza^on or system.' Included in the figure" are 
decisional, stagess indicating that the- principal purpose, of each^ fynctiori 
is to provide the necessary' informational .base in ' order .to -permit one 
to determine- whether to proceed, and upon which to base the. activities of 
the following step. The activities which go on at each step of the way are 
governed by the decisions which need to be made, and these dei;VLjs°ions in ' 
..turn define the Information which is required, and hence the inf ontiatlon 
which must- be generated. By adherence bo such a time-related Sequence, 
technical innpvat ion becomes the product of a deliberate and controlled 
process. * . 

« « • 

Critical to the process is a sharp sepaTa4:-t^n between thejbusiness of 
program (or product) des;ign and testing on the one hand, and the operation 

' ' ' ' . ■ • . . ■ - ^ o 

of .that program oi^ervice in the world-of-use on the other. 

Research and development is a high-risk venture, the risk being tlfat 
considerable resources will come to be expended only to have the , enterprise 
fail - a device, a process, or a program which emerges has a comparatively 
higK>iobabi'l^ty of provihg only- minimally effective, excessively costly, 
impractical of implementation, unacceptable to users, and the like.' 
It constitutes a gamble, one which few ongoing servite programs cap afford 
to pursue. Few operating agencies have adequate resources as they arfe to 
implement their missions; it can be argued that it is poor mahagemeiy; to 
allocate already insufficient resources', to commit; to high risk ventures. • 
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- Moreover, U_ is poor policy „ proceed „-,l„scall and operate a 
■ -rv-ice pro,ra.-prior to one's .iowledge .o ' Us' cosc.'and .capaMlUles 
Since once u 6eco.es. fro.en in rear and stone, onc^ people are.-.ired ' 
and. JO.S are,a. s.a.e. once fiscal-, personal and. procedural co,-,i..en.s 

become madei it will prove to be vpr,. H^^^^r- -, . ^ ' 

P ove CO be very difficult and costly to get it\ 

out again regardless of the<iledback infnrn. ^• ' ' 

* ' feedback information one generates concerning 

what it is that it can.^nd cannot dc^. ' 

in addition the probl^ .f trying to operate and test, so that one 
^ :..n;,^ise and^re^t, an ongoing s;;;rice progra, or system is a hit Uke 

trying to effect the redesien nf ^ i ~ ' / 

. edesign of a locomotive while it is steaming across 

•the countryside. Ongoing human service systems ' ' 

- ' - , a^'-vice systems are necessarily in a. 

constant ,tate oi a„., responding as tHe, ™st to ^.anses in co^nnit,' 
condition., client->lo„, personnel t„r,^ver, and fhe li.e, and/this 
Intro^^J^^^Wiiicatio^^s in progra. and procedure ajove and bi^ond those 
dictated by dJ^iH^d development' needs. 

_ Productive products and service^anCTe e.pected.witho.t careful ' 
a»ention to the design and testing pb.se. Because of tbe „any tbings „bicb 
~st be done in order to ensure tbat ^.i,al yieid bas been.e^racted 
prior to a decision concerning its „sts and effects,' it p;ov.s to be 
a .ulti-year. undertaking, and correspondingly re,ires^lti-year furi 
«itbout, 'adequate and continuing support .tbere' is too „ucb li^lib.-o. of 
failure.- it certainly cannot be sustain^^^e' basis of ^tbe feas^-and- 
famlne process of year-to-year individual grants. " 

Although tests of many sorts t^kp r,f. . ' " f 

y sorts take place at many junctures fhroughoul: 

the entire process ^hoV^■^l-4 i 

cess, the critical sets of tests in regard to the product ^ 

or pr.gra. are tied to the process of design, not to t/ o'perat-i^ii^l ■ 
pKase. This Is because ^.i^i f,..,,,,,,, ^^J^ ^^^^^^^^^^ 



necessary. The testing •of components* and/ procedures may signal the ' 
need for changes, a,nd "changes may 'be Indjlcated whj^h require "rapid and "'~ 

. sometimes profound 'alterations in the materials which are used, the 

[ - ■ . • "■\ . 

- order, in which procedures are followed, and .the lik4. Operating services 

• ■. • ' ■ S ^ ' ' V •' . - 

cannot function effectively under con^ilfions of day-to-day' chanee-^' 

^ ^^-^ ^ . \ w 

when program changes become introduced they must be .carefully del'ii^eated 
and planned and earn the endorsement of participants, ^us,' an .operational 
y^>%^ system inevitably responds to the necessity for change, much^mpre ' 
slowlyX To tie testing procedures to the oLrational pha'se ag we' have ^ 
beeo^cfoing in the human services sector is' to become enmeshed in a network ' 
, of compromises and accomodations to_service deliv^ery needs, which will' 
^invariably combine to dqfeat\one's purposes. ^ " • . . 

■ The capabilities: necess^r^or effective research and devel6pment 
r are very^4^if ferentf from thofee required, to manage and implement the products 

or programs, 0% t;hey have been developed, just as the- person Vho drives^ 
..^ an automobi;.e does not retire the same' capabilities as the automotive 
vy Engineer who designs and builds the vehicle, the knowledge and skill ; . 
required for the effective delivery of service 'to clients in need, and • 
the management and oi^eration of those services ip an ongoing agency, are ^ : 
of a very different calibre -than those required fUr research and design. 
UharacteristicaTlyv in their . traihing and back^rci^t^f their interesti( 
and prefereticesV anc^^ hence 1:heir technical capabilities, the two types 
of people diffei*. In research and development one needs access to design 
^ and evaluation technology, which is a body of documented information and 
skills, residing iri specially trained per sonneK whose principal comn4tment 
is 'to c-arry out- such development programs. Research and development 



cannot be effectively carried out oh a ^^^^i^^ basis by personnel whose, 
primary training and coramitment is to operat^l^B a' systemi administering 
and managing it, aAd^ engaged in the delivery of direct service. ' The . 
captain of a ship is not asked to be its architect .^and builder , although 
he. can and often does contribute, "to the prbceas.* ) 

Ab^ thus, part of the decision to proceed with a research and 
devei<5pinent phas^ involves an appraisal of one's capabilities for executing 
the R&D 't)hase. Characteristically, one needs an appropriate combination • 
' of facilities, .sustained funding, and personnel to carry out an effective ^ 
R&D effort, and the ability to expend a fair amount of money with no 
necessary guarantee ^of success. The necessary skills which need to be 
coalesced result" in multi-professional teams. Afe a result, in fields- 
such as agriculture, in medicine, or in engineering, only, a smalL pro- x 
portioA of iifBtallations have"* the capabilities to engage In* res^rch and 
development. The- typical farm^^ directed toward the pc^^uctjton of foodi 
and^ f ibresy^^henypical irfdustry toward g^ods and services, the typical 
medical ^cility toward the delivery of health care. Hi-risk, hi-cost 
R&D ac^tivity -goes on ^ri only a small • proportion of installations (such 
as Agriculture Experiment Stations, research hospitals, or in* development 
laborabories in sf few industries) where the necessary facilities, persorinel, 
equipment and resAirces have been co/centrated fc3r that purpose. 

Mapy more distinctions could be elaborated. Perhaps these are 
sufficient to indicate -that the process of human service program development 
and evaluation has proceeded very differently. Why is unclear although a' 
social histori^ might explicate the reasons why such a dif|^r^nt rout^ 
has beeii followed ^than has been 'the case with mor.t other efforts at - 
innovatioh. ' . . 
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Development of hi-effect lo-cost human service programs is something 
which will not come about by chance; we must alter our manner of .proceeding 
and. begin .to follow deliberate planful procedures to ensure that our program 
services move in the direction, of producing what is wanted. Complex human 
service programs cannot be expected to successfully emerge if we continue 
to evolve them in haphazard fashion, without adequate attention to their ^ 
Initial design. We must acquire the practise of conducting careful 
feasibility studies before we elect to proceed; we must organize oursel 
so as to carry out ef f ec tive 'R&D projects ^jefore we install servi(;es and 
begin to process clienTs. We must discontinue our practise of installing 
a program first, and then mounting an .evaluation effort to see whether 
what has been built proves to h^e any merit. Finally, efficient 
development of innovative programs is more likely to occur when resources 
are marshalled and directed toward thange within a well-organized and 
concentrated effort. One buys more development for a dollar when development 
is appropriately concentrated, than one can get when development effort^, 
are engaged on a piece-meal basis and extended over a very long period 
of time. 

Thus, continued pursuit of contemporary procedures of program eval- 
uation will not only prove to be excessively costly in the long-run, but are 
also unlikely to produce operational services and systems with the 
efficiency and economy which are sought. A shift of the field of human 
serviceH program development should be made toward an R&D strategy instead. 
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THE PROCESS OF INNOVATION 



Concept ion ^ 



[Teasibilitv Study] 
9-12 mos. 



Innovative generic design with prospects 
of success sufficient to warrant invest- 
ment in feasibility study 



A report justifying allocation of facilities, 
equipment, personnel and funds to a multi-yr. 



R&D project 



Research & Development 



I Design 
•| Trial ops 

[ Red esign _ i 
12-60 mos. \ 



[Red 



esign for consumer use) ^ 



[D i s tributicyi &^ D i s s em i nation]^ 



[Ins tall at I on] <f 




^ Evidence t'hat program is capable of producing 

\i effects'^n reliable fashion, within acceptable 

cost-parameters, in accordance with ethl^^, 
legal, etc. constraints' 



Program \M-rsion capable of being adapted, 
installed, together with all the means re- 
quired for its effective utilization 



Information display capable of providing 
potential users with necessary information 
to decideu,xwji^e'ther to adopt and install 



(Norma l_Op e r a t i o n s 
10 "+ yrs. 



Retirement ^' 




Produt^tive program, capable of being oper- 
; .ated anc) monitored, and with appropriate 
prbtedutes fol: its maintenance and periodic 



Costs and effects of program no longer com- 



petitivd with newer generic designs 
, plac<hnen,t. warranted 



re- 



Figure . The innovation process as it'il^j charac ter istically pursued in the 

majority of applied contexts. " The place of Research and Development 
in the process is represented in the context of a program life- 
cycle, arrayed along a time-line. 
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